This invention relates to multimedia networks, and specifically to the synchronization of multicast distributed media content when played by multiple multimedia devices, e.g., monitors and audio systems.
In some situations, such as at conventions and in exhibition halls, it is required to play the same video contents synchronized at multiple video monitors. Generally, it is also required to synchronize audio and video between separately located monitors and speakers. Current real-time media transmission technology allows transferring an audio/video stream over non-guaranteed quality of service (QoS) packet networks with reasonable quality and at a lower cost than provided by higher quality networks. Thus, there will be a demand for playing multimedia contents, delivered to a plurality of receivers over packet network, in synchronized manner.
FIG. 1 depicts a prior art multimedia content delivery system, generally at 10, which uses a packet network. A multimedia server 12 generates a stream of multimedia packets 14 that contain multimedia audio/video contents. Normally, server, or source, 12 generates multimedia packets at a constant rate (packet rate); e.g., one packet every 20 ms. Packet size depends on the multimedia codec, which may result in fixed size or variable sized packets. The generated multimedia packets are then transferred through a packet network 16, having plural routers, or nodes, 18, 20, 22, 24, 26, 28 and 30 therein, and are delivered to a plurality of multimedia receivers, 32, 34, 36 and 38. The mechanism:m for delivery to multiple devices is well known as xe2x80x9cmulticast.xe2x80x9d All the receivers that wish to receive multimedia contents from the server join a multicast group. The routers in packet network 16 are aware of the multicast group, and transfer received multimedia packet to multiple routes. Each receiver is equipped with a speaker and/or a video display, and each plays the contents in the received multimedia packet in real-time. The multimedia receivers may buffer received multimedia packets for some duration in order to absorb the variance in delay caused by the network, and to playback multimedia packets in right order.
In conventional packet network, such as an IP network, packets are queued in each router. How long a packet stays in a queue depends on the amount of traffic that a particular router is handling. Hence, each multimedia packet delivered to each multimedia receiver has an unpredictable delay. As a result, the audio/video contents may be played at multiple locations without synchronization.
As previously noted, real-time multimedia transmission technology currently allows transferring an audio/video stream over non-guaranteed quality of service (QoS) packet networks with reasonable quality. As mobile Internet access increases, with its data transfer over bandlimited communication channels, such as wireless data networks, the technology to transmit real-time multimedia packets requiring minimum bandwidth becomes essential and of increases importance.
A packet-based real-time media transmission system is illustrated in FIG. 2, generally at 40. A multimedia source, or server, 42, periodically transmits multimedia packets 44 having audio/video multimedia contents to a packet network 46. In packet network 46, each multimedia packet is routed by a plurality of network nodes, or routers, 48, 50, 52 and 54, and delivered to a receiver 56. Each multimedia packet includes timing information that is used to set timing for playback at a receiver 56. The multiple routers and receivers of FIG. 1 are not shown in FIG. 2 for the sake of clarity.
Packet network 46 causes a: variable length delay in the delivery of each multimedia packet. In addition, network 46 does not guarantee the delivery of the multimedia packets in the same order as transmitted. Thus, a receiver is normally equipped with a buffer to handle delay variances and/or disorders in multimedia packet receipt. However, due to the real-time property of the contents, the receiver will discard a received multimedia packet if the delay in its arrival exceeds a certain value, or if the multimedia packet does not arrive by the time it is to be played.
FIG. 3 shows an example of packets received and played back by the receiver, where each packet contains the timing information. Line I depicts seven multimedia packets transmitted in sequential order by server 42. Line II depicts the order of receipt of the multimedia packets at a receiver. Line III depicts a play order by the receiver. The figure depicts that the multimedia packet with timing information 3 is discarded due to an excessive delay. Basically, the routers in the prior art packet network are not aware of delays in packets transmission and receipt, nor are they aware of disordering of packet transmission. Thus, each router re-directs every received real-time multimedia packet to the next router or the receiver, even if the multimedia packet may not have any chance of being played at the receiver. If a physical link between some two routers, or the physical link between the receiver and the adjacent router has limited bandwidth, the transmission of the multimedia packets which will be discarded at the receiver possibly wastes substantial portion of valuable bandwidth on the link. Delay and disordering is illustrated by the variable routing of multimedia packets from router 48 through router 50, router 52 and router 54, or directly from router 48 to router 54.
U.S. Pat. No. 4,506,358, for Time stamping for a packet switching system, to Montgomery, granted Mar. 19, 1985, describes a method of measuring the network delay in a packet network by including an information field in each packet that indicates the cumulative network delay, time stamping each packet""s arrival and re-transmission in each network node, and updating the information field in each network node for each packet by adding the delay occurring in each node, i.e., re-transmission timexe2x80x94arrival time.
U.S. Pat. No. 4,530,091, for Synchronization of real-time clocks in a packet switching system, to Crockett, granted Jul. 16, 1985, describes a means to synchronize a remote real-time clock to a standard real-time clock via an X.25 (CCITT) network, but does not teach or suggest how to synchronize a multimedia stream, and does not consider network delay caused in a packet network.
U.S. Pat. No. 4,918,687 for Digital packet switching networks, to Bustini et al., granted Apr. 17, 1990, describes a method to avoid consecutive packet loss by clipping voice packets whose contents indicate whether the packet is eligible for clipping, if required, in a node queue within the network.
U.S. Pat. No. 5,432,790 for Method for allocating internodal link bandwidth in a packet oriented communication network to guarantee delay quality-of-service, to Hluchyj et al., granted Jul. 11, 1995, describes a method to allocate bandwidth on internodal links such that worst case maximum delays and worst case average delays are guaranteed while providing statistical gains in maintaining predetermined end-to-end delay QoS.
U.S. Pat. No. 5,444,709 for Protocol for transporting real time data, to Riddle, granted Aug. 22, 1995, describes a method to maintain the isochronous nature of the real-time data transmission by discarding data that is not timely received.
U.S. Pat. No. 5,450,394 for Delay monitoring of telecommunication networks, to Gruber et al., granted Sep. 12, 1995, describes a means to measure a round-trip-delay of network transit time, however, the delay which must be known for multimedia transmission and synchronization is a one-way network delay.
U.S. Pat. No. 5,557,724 for User interface, method, and apparatus selecting and playing channels having video, audio, and/or text streams, to Sampat et al., granted Sep. 17, 1996, describes a user interface in a computer system to playback multimedia streams.
U.S. Pat. No. 5,594,660 for Programmable audio-video synchronization method and apparatus for multimedia systems, to Sung et al., granted Jan. 14, 1997, describes a means to playback an audio and video stream synchronously to a single network-based receiver.
U.S. Pat. No. 5,598,352 for Method and apparatus for audio and video synchronizing in MPEG playback systems, to Rosenau et al., granted Jan. 28, 1997, describes a means to playback an audio and video stream synchronously to a single network-based receiver.
U.S. Pat. No. 5,623,483 for Synchronization system for networked multimedia streams, to Agrawal et al., granted Apr. 22, 1997, describes a means for a receiving device to determine the total end-to-end delay from a statistical network delay distribution and the synchronization of the two media streams from different sources through the provision of a synchronization clock located in each server and in each receiver.
U.S. Pat. No. 5,629,936 for Control of consecutive packet loss in a packet buffer, to Lee et al., granted May 13, 1997, describes a method for dropping packets to control consecutive packet loss based on a desired packet gap.
U.S. Pat. No. 5,675,573 for Delay-minimizing system with guaranteed bandwidth delivery for real-time traffic, to Karol et al., granted Oct. 7, 1997, describes a bandwidth allocation mechanism based on individual guaranteed bandwidth requirements associated with each traffic flow.
U.S. Pat. No. 5,682,384 for Apparatus and methods achieving multiparty synchronization for real-time network application, to Zarros, granted Oct. 28, 1997, describes a means to playback contents from multiples sources synchronously to a single receiver on a network.
U.S. Pat. No. 5,768,527 for Device, system and method of real-time multimedia streaming, to Zhu et al., granted Jun. 16, 1998, describes a method to recover lost packets by retransmission.
U.S. Pat. No. 5,802,060 for Data network and a process for real time transmission of data packets in a data network, to Fischbach et al., granted Sep. 1, 1998, describes a method to prioritize packet transmissions for stations whose network access is controlled by token exchanges.
U.S. Pat. No. 5,815,634 for Stream synchronization method and apparatus for MPEG playback system, to Daum et al., granted Sep. 29, 1998, describes a means to playback an audio and video stream synchronously to a single network-based receiver.
U.S. Pat. No. 5,844,600 for Methods, apparatus, and systems for transporting multimedia conference data streams through a transport network, to Kerr, granted Dec. 1, 1998, describes a means for audio/video conferencing over an ATM network. Audio streams are combined in a network-based mixer and distributed to terminals. At each terminal, the received audio and video streams are played synchronously, however, there is no provision for synchronous play at multiple terminals on a network.
U.S. Pat. No. 5,874,997 for Measuring and regulating synchronization of merged video and audio data, to Haigh, granted Feb. 23, 1999, describes a means to playback an audio and video stream synchronously to a single, network-based receiver.
The Internet Engineering Task Force (IETF) publishes requests for comments (RFC) and comments filed in response thereto.
RFC 1889: A real-time end-to-end protocol utilizing existing transport layers for data that has real-time properties.
RFC 2508: Compressing IP/UDP/RTP Headers for Low-Speed Serial Links
RFC 1144: Compressing TCP/IP Headers for Low-Speed Serial Links, Similar to RFC 2508, but for TCP/IP packets.
A system for transmitting and synchronizing real-time multimedia content includes a multimedia server for generating a multimedia packet having multimedia audio/visual information therein in a play sequence, and for transmitting the multimedia packet; a packet-based communication network connected to the multimedia server for receiving the multimedia packets therefrom; the network having multiple routers therein to route the multimedia packets to plural destinations; means for inserting total delay information (TDI) into the multimedia packets, wherein the TDI includes total end-to-end delay (TED) and cumulative network delay (CND); and plural multimedia receivers for receiving the multimedia packets having TDI therein, wherein each multimedia receiver includes a buffer for storing received multimedia packets, a sequencing means for retrieving stored multimedia packets from the buffer and sequencing the multimedia packets in the play sequence, and a playing mechanism for playing the retrieved, sequenced multimedia packets at time TED after the multimedia packet has been transmitted by the multimedia server. A method of transmitting and synchronizing real-time multimedia content includes loading multimedia audio/visual into plural multimedia packets; inserting total delay information (TDI) into at least one of the multimedia packets; wherein TDI includes total end-to-end delay (TED) and cumulative network delay (CND); transmitting the multimedia packets to plural multimedia receivers over a packet-based network in a play sequence; buffering the multimedia packets in each multimedia receiver; recalling the multimedia packets in play sequence; and playing the multimedia packets at each receiver at time TED.
An object of the invention is to provide synchronized A/V output at a plurality of receivers over and multimedia carrying network.
Another object of the invention is to prevent transmissions of any multimedia packets which will not be used by a receiver on the network.
These and other objects and advantages of the invention will become more fully apparent as the description which follows is read in conjunction with the drawings.